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DmaTt Sirsi 

I am pleased to submit to you a special Interlia report 
of the Katloml Advisory CDuncll on the Iducatton of 
Disadvaiitaged Children.. 

The CounQll la reporting on the evaluations launched iri 
1973, by both the mm Office of Education and the 
Natioml Institute of Educatloipi. Both evaluatiqns will 
eKamine the operation and eff ectivenese of programs ^hlch 
Local Edueatlonal Agencies (IMm) carry out under Titila I 
of the Elraentary and Sricondwy Education Act, the Title 
providing for eoTOpensatory education asslstaiice* 

We ainearely hope that our views and racoimeiidatlons will 
proye beneficial. 




O^en P* Peaglax 
Chairman 



The Jresident 
Tha White House 

HonOMhlt Nelson Rodkefeller 
PreBldeat of the Senate 

Honorable Carl Albert 

Speaker of the House of Repreaentatives 
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FORETORT 



In 1973-74j the Elementary and Secondary Educatioii Act 
(E.SsE.A.) undatwent broad^ detailed review*/ by CorigreBs. The 
result was Public Law 93=380, the Educatiorial Amexidinents of 
19745 enacted AugfJst 21^ 1974. 

Certain aeetiDns of PL 93=380 fliaadated that the National 
liistitute of Mucatiori peTforia a major stiidy of the operation 
and admlnls trattori of Title I, E.S.E^A.. compensatory educatlDn 
programs * 

Bectlom 821 of that Act also stipiiLated that tlie National 
Advisory Coiincil on the Education of Dlsad\rantaged Children 
(NACEDC) "shall ad\risa the Institute with respect to the 
design and eKacution of imh study. 

The Of flea of Iducatlon also racelNred a charge In this 
field/ Secttom 417A of tine E.S*E*A, Amendments of 1974 called 
upon the Conmiss loner of Education to present to Gorigress not 
later than Novetaber 1 of each year comprehensive evaluations 
and surveys relati.'^re to the Act. Section 151 of tha Amendments 
also cominlssiotied the Office of Education to uiidertafce a broad Title I 
evaluation* The Commiss loner of Educations It w&m stated, 
^*shall provide for independent evalTjatlons vhlch describe and 
raeasura the Impact of programs and projects under thti Titles" 
Title I. 

This too became a subject within the ady^isory scope of the 
NACEDC not only because of the Council's broad enabling mandate 
but also becaiise of the two studies ' coincidental tlmlrig and 
common aubjecti a major Title I, E.S.E*A» evaluation, 



1 

6 

EKLC 



IMmODUCTlON 



The repQft which foLlow^s presents a review and critlqut of 
the procasF, steps and arrangenients involved in the how on-gping 
major evaluations of compenaacory education by two agencies Linked 
to the Department of HeaLthj Education and Welfare, the National 
Inatlcute of Educatioii and the Office of Education* 

iiils Ib a case study of the birth, growth and problems In 
deeign of a major FedeiraL evaluation, 

^ The Elementary and Secondary Education A.ct is a decade old, 
Congress J In 1974, mandated a major evaluation of Its Title Ij 
by NIE, while adding language and emphasis to the on-gclng eyalu-- 
ation reyponslbllitles of OE (see ^ppendljCj herein). 

rha Legislative and Exacutiva branches both face the difficult 
ta^k of fine-tuning the adralniatratlon of not Just one but hundreds 
of social prograins enacted within the past decade. This fine-tuning 
requires the strongest body of evidence and insight, making possible 
basic alterations andi when indicated j terminatidn of programa; 
Evaluacion as a process Is growing in importance as mora and mora 
attention in domestic programmlrig. focuses upon assessing and 
irnprovlng results^ aspecially cost-^ef f ectiveneas , 

The CQuncil's report outlines weaknesses in the way these 
legislative and executive responsibilities nQ%^ are being carried 
out with respect to one prograni-s evaluation , but with implications 
"writ large'' for the legislative and eKecutive process aa a whole, 
I'he leesons to be learned^ we suggests Include theser 

0 , ITiat the Congressional practice of enacting much uiajor 
legislation lace in a fiscal year does special danvage 
to research' proj acts whose design and contracting then 
must be telescoped, A better effort must be tnade to 
- accord research a realistic time framap 

o That Congress and the Adjninistration need to provide 

Iniproved overslgjit disciplining and cQordinating^ - — 
separate agencies^ tendinclea to embark upon connected 
research without coordinating arrangements * 

o *That the practice of assunilng that varied agencies con- 
cerned with a given problem naturally will coordinate 

^ i « 
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carriis too far a naive falth_ in good administration ._Ths 
Execattvk nusC examiria tHe prptlem of cobrSinition with more 
reallsffi and commitment * Its coordination is now too of ten 
''aftsr the fact**, ritualized and weak* 

0 Most iittportaTit of all, in the design of evaluations ^ there must 

be fi parceptlpn of the relationship of programs to problems 

that goes beyond nara maasuremftiti of a program while in progress. 

What laeting Impact does it have? WTiat is its relative value? 

Thus, the/l&iues are far broader that merely the obvious posslbllitieB 
of overlapping objeGtlves and of adminiatrative shortcomings; the issues 
involved alsg inaluda Congresaional expactatlons (and timetables) > inter-- 
agatiey ccordinationi the role of Advisory •COBmaissions and Councils, 

The analysis which follows documents the need for modifications In 
the two programa of study ( one by OE, the other by NIE) now underway 
eacamining the effectiveness of Title 1, E,S,E.A. 

fh0 maJoT points ape^ 

9 That bBtm^n OE and NIW there i# a aoTmon definition amf stan^ 
dca*d of 0ff£ativ£neB8 of aomp&n&atoTy mduaation progrmsj taking 
into aonBidet^ation all the vai*iaiZ&a in a sahaol dietpiat^ 

o That b0ti?emn OE and NIB there be a Qommon underBtanding rmgax^ding 
wha^ the loocxl &duoation agenQi&B (LEAb) should bm es^eated to 
do cmd a(^aQnjpli3h with their reaoyramB ^ For emampte^ if reading 
cQtd math havB ctppTOpriate high priority in baeia eduaation^ 
BUppoTted by statm and toaal funds ^ then what is the rdl& of 
aomp$n$atory &duaation ? 

o ' That the Off atudy dBsigjt in particular be revmiped to provide fov 
aSB&BBmeYi't ov$r a suffioiently long interval * a everal ye^s 
ZonpB^ tha n now ptmned - to really t&at effiaaoy^ lasting itn-- - 
paot and eff&ativmneeQ - aBaeBaing etudent performance some 
period after thei^ Title I prog^m exgosu^e* Lasting irr^aot is 
the be&t TneaBuTe of vatue, 

o That tohgitudinal Btudiee ehould^^^ Teguta^ pradtiae 

in Title J mad not a rare event* In faatj the tedhni^ue is bo 
valucblm that it Bhould be emplQymd by Btates and at least the 
largex* eahoot diatriatB^ albeit with smalter amtptea and tower 

QOBtB* 

0 That both Btudy dm&ignQ^ but eBp&oiatly the NIE d&Bign^ must more 
aatively ec^loTe altemativeB imtead of foausing (ae they do now) 
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larg&ly upon thm rmfimmen'^ of pr&&&nt arrangem&ntB aentered 
ox'Qund the sdhoot. What non-^BGhoot proa&38m3 would pTaduae a 
b&ttsr Bduaational dividend fop $300/Btudmt ? 



A word about the Couiicll's position. The studies involved now have 
begun — that is, contracts have been, let, and actual work undar contract 
has started. Yet the Council has serious concerns regarding the deaign of 
the present evaluations* The NACEDC recomraendt that many of the questions 
raised should be takan Into account by those directing and loanaglng these 
studies as wall as by higher officiala who approved their activation and ferttp 
For money is too precious and tltne always too short to permit theie deficiencies 
from happening or continuing. 



PART 1 

THE STIJKIES' BACKGROUND, DEVELOPMENT, INITIATION 



Briefly tracing the anatomy of thega studies helps provide pers- 
pective and also h#lps illuminate some of Che problams Involved In 
Titla IjE.S.E.A* m^XmtionB* design and Implemotitatiori, 

Compensatory education became a major place of social legislation 
in 1965. In BUbBB^mnt years, funding slowly expanded. Bo^ too, did 
concern over effsGtlvtnass. In 1972, Congrass responded to Administration 
suggestions for m£mm^ and adopted the strategy of focusing available 
r©aourcea In thope pvtoaty schools with aubstantial proportioni of dis- 
advantaged childreni. The Department of Health, Education and Welfare 
sugges tad that a $300/pupll target sum was a level of funding which 
the then-axis tlng^ and admittedly IncQnclusive research suggested aa 
the "critical maae"' of aid for compensatory education at which there 
was significant impact.*. the size, scope and quality to aake a differ- 
anca. Beyond thli giineral funding level. It was suggested that any 
marginal benefit tXQm added money per pupil was not cost-effective. 
Of the total resourci^a available, now approxlmting $1.9 billion per 
year I ovar half In ^pent In emphasis upon the school yaara kinder- 
gar ten through third grade. Almost all the resources are applied 
kindargarten through alghth grade. 



Proprapi Assumptione 

One cannot avaii:i^te a program Indapandant of that program's con- 
textr the problam ftddressed, the assunqptipne made, and so forth. When 
a program Is desl£«i^4^ Cha EKecutlva, Congrass and Qthtrs Involvad make 
assumptions detawftlwfcig the basic form In irtilch the program will ba 
iaplamentad, assuntpt^loiui ihaping how the funding will be determined and . 
dlsbursadi 

What are the b^aic aasuraptlons behind coi^ettsatory education, 
principally Titla ir v^ 

marizatlon of legliJt^tlve history and claearooa practice over this 
deaada, certain m J or afisuBaptiona are obvlousi wa believe j from the 
form in which toplTOmtation has taken place i 

1. That the alowat rata of learning on the part of children, 
when manl<#.afc^ raprasents disadvantage on their par^^ 

2 , That the pchoola in which these axe a prepondarance of young 
people perlorwlag below norma have not tried all they can. 
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3, That the slower rate of learning on the part' of educatiouaily 
dlaadvaiiCagad children represents a conditiori that school systems 
with focusedi financial assistsiiGe could subgtantially' overcome^ 

4, That a preponderant proportion of disadvantaged youngsters live in 
areas with lower financial reeoiarces* Diaparities of both home 
and school have been interlocking i (The disparities are net all 
negative* In many ways disadvantaged children grow up learning 
special skills - e*g,, survival skills - upon ^hlch schools need 
Co capitalize. ) 

5» That the schools as a delivery sygtem represent an appropriate and 
affective place in which to ovetcomi the weak learniiig rate that 
tends to be common among the disadvantaged, 

6p That the regular curriculum aiid school budget could not' 'adequately 
provide for this problem in echool areas where disadantage ^as 
prevalent i That a major solution to these special needs required 
specials concentratedj cofflpeiia^atory educational approafihea tacked 
by special federal funding but largely designed by educators at the 
local level, 

?• That special federal fundliig \70uld spur the development of nmy and 
innovative techniques to accompliah what conventional education had 
failed tp accompHahi That local schoala knew their youngstars' 
problems and needs best, and i^ith esctra money could institute 
advancefi to help them learn bstteii 

8. That increased educational attalnnient would lead to increaied em- 
ployability and productive citizenship • 

So came Title I, E^B^E^k,^ a decade ago, * 

It was one of the major products oC the I960' as that era of social ferment 
and v^ide-- ranging initiative seeking reBiedies to the discontent and gaps between 
tiava^s and have-^not's in our iociety* From the Voting Rights Act to Head 
^ StartV the— era sought to attengthen the performance of eKistlng Institationa ,- - 
to hiiring the poor into the political prpcesSj and to remedy the causes of 
discontent including little dividend fro© conventional K-12 education^ 
handltcapping the disadvantaged in eraploy^ment and socio-economic acatua* 

Many of the social programs enacted In the 1960 's mainly were "envelopes'* 
for a federal investment in a problem ^.i.^, £n many respects this applies 
to Title I* Seconds and with special relevan-.e to Title I, even educators 
themifelves always have lacked precise cieicis for predicting what learning 
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dividend, if any, should flow Etom m glveh Imml of finarialal aomiiMxt 
And probably tor good reason: for to effeot such pred Ictab ill ty would re- 
qulra a centralization, atandardiaation and aven reginientatlon of education 
out of keeping and out of plsca la the Ajneriaan cliaracter and Bcene. 

EmluaHm is not a my out of bhm ^ndlem variety in ffffsatiDmess that 
local Uademhup pvoiimm, becmme that mdZsss varisty is samntial in om> 
^yatam. _ In faat^ d^ifemnms in baak^romd and davmlommnt unier'am mv 
asownption that eduaaUmat attaiment Qm_ aonpavahZ^ be asBeased. 

This makes national, asaeasnent of tha oparatlon of a program such as 
compensacory education very difficult and, moTa imjoi'cant, frought vlth 
the danger of producing gindings hcraogenized lato £ nilsJaadtng national 
set of statistics... avepagas and norms baacing little raitvaace to aach 
parcicular case. The nieaiuremants applied have widely mrying lelavance 
notwithstanding the sincere efforts by proirani admlaiatrator s to develop 
objactive appraisal Instruiiieiits C'culture-f cee tasts", ate) , 

We have similar r-esafvatlons about the efflcacr of evaluatdons of 
Che administration of progtaina, the special aoiphasis of the HIE progiam 
of studies. Every evaluation of coiapen aatory aducation to date has ladl- 
cated that there have been insignificant or only- BllghtLy slgiiiflcaxit 
gains. There also has haen difficulty in o&wpaTatdve neaaurement beyond 
Che local level. 

The compensatory edacatlon ritle of the Eloinaritary and Secondary 
Educacion Act in fact assuities this, by Ics basic feature of rnaklag avaUatle 
federal funds to local prograitis meeting certain guidfllines criteria but 
essencially local and cherafore tending to always ba Linlque la desLan and 
character. 

What this suggests is that longitudinal atudles assassing program 
efficacy ( studies in vhlch tbe educatioaal lavel of a youngster accordtng 
CO some grade level criterion deternilned hy tlie Ll^^ is coinparea agaiiiat 
the level of that same youn,|ater last year or the ^ear before) do nuke 
sense. Studies trying to discern bast practices ( by comparing pi'ogress 
across systeins and betvaen gtoxipi) ara li-kely to la far less mcanlagfua. 

"--^^'^d yet, progra E caiinot ha presumed to worrTiiersly " 

because their alms are desirable and -we vouli lilte chein to u'otIc, me 
entire public process should have an interest: in m king oliIj tiie coimnitments 
in this^ broad field proveci to prodaee lasting heneflts, not cenporartlv 
"hypoed" change or a mere d«inonstra tioii of sociQcal concern. I'ha piobieiii 
of effactlvely helping thfe dltfadvaiitaged to succaed is too vitally Important. 

Within the limitations these thoug'iits sugiest, there in enongh avldctice 
in existing research that educational fundliig level hue little to do wltii - 
achievement, -to warrant thorough -going testing of basic proiram assuniptioiis ' 
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behind T:itle I, E.S.EA. Educational atcJlrxniEint da a function of-. 
;,so'ciQ-acDa:omic factors in which "ths scIiqd 1 is part of a far iarger^ : 

CQpjiip^BK sa.L-ot inf lufenc&s In cha baekgTouJi^ and asiviroEiiiieTit at 

youag p^op le^ Sck«3lats in tha f ieLd alao remind mis of th^ rMge 
of Indt^idiiaX d J.ff erfeiice.s involvad. Lii a3iy sttiool tlie^e "t:ends to 

a .w.lja Tarmge of ability and learPtnLlj ^ point arguing £or proceases 

Itidt^idTial as pcsaibLe. 

Ol^aciyi this is a field in v^liicli th^re are sir loui iitnitations 
to tbe ,appiicstioa of social sclenaes-^ ciiartent conJlderice th^t 
it Can... ^uamtify inewtiig£i^^^ the mfeasuLtag o£ ttie eff 4ccivent#s of 
a process ^hich ^ w^tiea Dtxe anQlyzes dt^ Qti^ fitids Bia^lly dip^nd^nt 
u^oa maxiy ^etioraal^ t3idL=vidutl v^rlabL^Bj. ttte niost b^aic of ^hich 
■ in: tlils ca^e is the tjidlT?id.ua3. teaatitr , 

The t^dde diveyslty^ — in fact, paaard^atlon ~ vlewe about how 
tc^ isiprovfe tbe effect ivaMsa of edacatioa of disadvantaged ct^iUrea 
c^rtaiajy niakes it cL^ar tKat there ca* rxo easy g^neraaii^ticni or 
conciusdQii^_ , oo cla^r quantification of da^ta that ^Uppott^ anj 
pa^rticular ap^proacti coMLusivei^ Thm pcor f lov of Icxfo^ffltciom 
iTL tHe ©ducat ional sy^stein C up and do\rt ) fuittlier uaJenioe^ the 
va lue of a msearch pjoduat , 



R^-qul^rettiBati of Effective E^amacio^ 

To evslu^tfe meias t© t€St ef feeti^eneas and efficiency. The 
re<|ul^ramenei effective a-yaruatiort tBat ar< poljcy Le-vaL t^^ther than 
laeaeLy per£ociiiati.ce Uvai ara i . 

.1. The e^aluatiori must taat the picog^atn 'a basia asflumptt-ons* 

No av^alu4ti.Qii of a poLic>r level o^daj can ba Judged p^ffectlwv 
unless constructed arotind i te^t. the asBuniptiorxi b ehind the 
inter^an tiopna g Im inaiidate^, It is the ciatc pi a^^Liiation 
to tast^ assumptions about wl%at «Q:tks and ^hy ^ It ^h^^id help 
id^ntif^ co-rrect jassuinptlflns . So^ &vaLuatioi5^dfill4t\ mat pToduca 
an organigfed snaJ-^sis to produce the data ftok which can be 
sKtrapoIatfed quanpitatlve ar^d qiiaXitst^ inMgfita , v^^lidatlng, 
rag uting , or rnodifyiag tbi o|ei:ati\?e sat aiaMpttp^s be- 
Tiind la^* arLd -practice* 

X Eva.J.ustlcn mmt procaed fTOrn ciear definition of the yroblara 

iidd-r^sied aaid tli^ p^ogMia tti tijrvnc^tlon %md^, Beflflltioii Is vital* 
lacking dt^ oae cantiot ba certain that i&fha t ±b tPilng maaured is, 
in fact, th^ response to the ori|l.nti oT^^ It t^tght merely 

te tli^ pytiia-aiu raclanale iwhlch, like a jro tectlve coa*! kesps many 
yro grams "w^^mi" ©van after v^^lKi^stancing o-riglnal jurpoaes. 



. .. One alao cannot be siire, licking cleax 'def Inltlonp that the 
meaaurements which evalutatlon axrives at really relate to 
OT provide relevant faedbacTt in tema of Congresgioml intents 
QT represent aigtiifitanc Insights. 

3s Everyone who is key muit mnt the evaluaCipn and woik at j^tei ng^ 
It TOrk , whether Congressional ComltteeCs), OM , OE/HEW or NIE- 
and local level education adnnlnifitrators * 

4, Regrettably, there can be no Dbjactivei meaningful ev^aluation 
of education coat ef f ectivenesap beca\ise there is no accurately 
testable corralatlon between expenditnrea to pyoduce the inter- 
vention and program lmfi_lMaiitatlon ( clear* visible evidence of 
acmethlng like $300 In value showing np ^ value of services, that 
is) , pjco_gggitt _Qu^cQme ot_ perf orroance and finally, progj^am Impact « 

All are impcrtant. All have significance in this preeant context . 

These are the problems of approach we believe must be taken into account 
Itl any prof essloTial , objective e^ralnation of Title I coapenaatory 
education. The Council maintains tliat in any professional evaluation ^ 
of E,S.E*A. ^ - . 

o nia evaluation must bring precision tc the correlation 
between eKpendituresj Inflple^entations outcome and ispact 
which in well-^administered programs should be on-golng| 

o Hie Vest measurement of eff ectlveneas ii longitudinal ^ 
that is, ovar a long enough term to ref lacc adeqv.ttely 
whether .the benefit and gain last ; 

o To provide the most valid test of opeTattve asauinptionSi 
the evaluation must openly explore and wigh alternativeei 
(lor in the search for the "best "fruit'% it is not enough 
to merely compare apples) . 

The Adviso ry Couneil'a Concerni Regarding the OE/nIE Studies 



In December 3 1974^ NIE published ita Research Plan for Compensatory 
Idncatiori; by March, 1975j RFPs were being Isstied in a series now (as of 
December^ 1975) nearly complete* The 01 tlae table was comparable. 

On January 31 , 1975 the Council reported to Congress and the 
lacecutive tl\^ various queationa and reservations it had about the way 
in which the OE/NIl programs of study were taking shape. 



Ill the fall of 1974, the National Advisory Council obsarvad that 
OE and NIE iv^ere making ready research plans, that coordlnatlDn was t^eak, 
and tliat tl.ci CQUnctl itself had been inforiaad but ^as not being closf.ly 
coiisulted. me agencies seeaad to* assume that their studies wre mo 
dtffeTent la design that the value in Inter-agency ceiisultatlon waa small 

The Council took the initiative to ask for clase CDnsultatlon* 
Howevarj for many months the arrangements proved weak, usually conaiating 
prl«aril7 of the agency advlelng the Couiicil when It tras about to take 
a course of action the agency had already decided upon. 

At the mmm timm the Council found In TICE and other bodies 
improved means of coordWatl6n and impact for Its vlewa* 

o The .Council 's conceru about coordination has resulted in the 
formation of an appropriate FICE working group | as one indica- 
tion of progress In cooperatton* this working group no^ has 
agreed to a sharing of Instrumentp before going to forma- 
clearance stage, factlltFtlng cooperatloni 

a nmtm are now monthly meetings between oB and KIE staffa in- 
volved in these s,tudt€s» ^^Ith the NACIDC Invited to be present, 
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PART II 

COMNTS REaARDlNG THE Nil 
"RESEARCH PLAN i CO^ENSATORY IDUCATION STUDY'' 
AND THE RFFs WHICH FLOWED FIOM IT , 



.Totally apart froin queatloftt of daslgn^ the nechanies of handliiig 
th^ iaitiatloiL of wida--rangin roajpr leisaxeh within a very limited 
tins firane clear Ij^ craatad problems for the ageiicles Involved^ espaelally 
.Nil* For the ME 'approach heavily dep^nded-^^ u^ gaining school distTtct 
cooperation'^ a difficult matter, A typiaal ff^hool diat^ot ie asked to 
ao'ope^atB in seme SO Burvmya and stuM&B cmnvmlty by MEW atam^ 



DlatTlct Salectloa ' 

A. word la In &tder abo^ut the way In which NIE developid its aelwtloti ^ 
of districts te be iavolved In tw of t^a najor stmdles oiE its ragearah 
program* for bcth prQcesaei appear weak* 

For one major phaea of Its study ^ the dcmosstratiofi p^ojeatSj N3E 
sought noBilnatlonM through the atata Departmenta of Eduaation aa 
.y eKtended process which nraant that the feedback formalizltig a diaUrict^s 
agreeiaent to aooperate and participate xoay hav^t taken longer thart the 
tine Va^^llable * ^ to amd ability to nominate ^ a atatep 

■ ; d^peiided^^ politics as well as atility to a^t in tlae, Houston 

^ itiidy /because a Houston school jofflcijLl^^^h^ 

; dur ing meet lags in Washiiig tea and suceeflsf uily asked hat s tate to pro- 
■ : ■ ; oess the noialnatioa, Net^ York^ ChioagQ and Lob Angeles also wanted to 
partlGipatej according to NlEj but thsli statss did not a&alnp.te thia. 

Totiii iioml were far below expec-cat Ion* Out of 50 states 

that could have nominated only 27' discrlcte nQminated^ 

^ from which Hlg then selected 16 proposais as the "tegt wittaa", Ifte 

result I we suipacti Is that tha diatrlcts in this phase of the atudy 
: i^ere not as representative as they should have been* 

-toother major phase of the study is district Survey a large eurv^ey 
of practlcas in corapenaatory education la 105 districts* This too has prob— 
'. lams of repreientatlv^eneEis, To dra^ this sanplej NIE f:lrit divided the, 
^ nation's 16ppOO school districts in thlrds^ty ai^e. Thm 5,3J5 larger 
: :r :dlstrlcts were than assumed to be largep the neitt third toedium,.. and so forth* 




: HIE then proceeded to draw equal samplts frow each of these thirds, to 
comprise the final 105, 



Hacause it has net been provided with the necassary information^ 
plE is keeping confidential the list a| the 105 dlaCrlcts involved " 
Jin the Intarest of helping assiire' their coopetatloti) , the CoUnc-il doee ^ 
mot know the degrsie to \^hich the NIE-a«lac ted sample of diitrlats is 
effectively representative. NcmethelesW, werely cti the hasls of NlE^i 
sampliiig method, ire have a serious question* There is a mjor question^ ^ 
as to the kind of representatlvenesa that would have been best/ NIB 
points out that its sampling technique aought to produce a saraple repregen 
tative of the natlcn*s approHlniately i6,000' school dlstrictsi In this 
approach, the dlatrtcte iii the largest "third by student sige would 
comprise a thltd oi the sample* 

Thm Council advised ME that far preferable would be sampl lug repre- 
sentative not of districts par se, but ^lal student population and program 
cost. The very large districts do in tmt have a proportion of the 

Title I students chat is far beyotid their numbers as dlstrlcM, 
aM thiy have special administrative problems t^tth their Title I 
prograBQS alsci. 

The poiat is not minor, NIE deemed that a district warn large if 
It hart over 18,000 students, as It turned out| the Gouncll deeins a 
large district to he more o^ the order of 10O|OO0 students, 

i 

MIE Research Oh j actives 

. .NIE - a . Reaearch Jlanr Compensat or y Education Study (p,2) diescrlbes - 
the ohjectivea of Che research in language which Includea: 

This study's proper concern Is with |he adequacy of educatlomal 
programs provided to students* It w%Xl SKamiina prcgraiia and 
look for identlfiabla factors in fundings organlzatlori and 
instructional methods that BKplalii success. 

While ^e will deal later with the my Nil has assiEibled cgoperattng 
districts , It Is worth noting here thf.t identifying and explalnlfig 
success ( requiring focus on programs known as suacessftil) probably 
would have led to an entirely different approach from that which KIE did 
in fact follow in asking state Bepartttitnts of Education for nominaltlonsV 
Tor in the practice followed, cooperation tended to be the dominant keyi 
not knowxi success. . ' 

The 'NIE Plan warns of difficulties facltig this research which ^ are 
inherent In the edacational field* 
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Lacte of uniform standards of assessmeiiti tliere ara so many 
variables that tio> effort at unifoTnlty; could be fair. 



a Shortcomings of existing data; wliat data does exist on 

ediaaational progress la sketchy md uniellallaj there is no 
uniform testings and NIE will not provldt it^ NIE warns of 
the difficulties faGing their aCatatoriLy required research* 

Ihe NIE Plan recogtiigas that thare is a gMat degree of local 
autonomy in development of the Title I B,S,E*A, ^progratt* rharefore Nil 
considers It Important to axaminSj in a raftga o£ diaCrictSs what 
goals each local district has set for the Title I program ( e,g- , 
special emphasis on bl--lingual InstructloEi In momm areas ^hat 
relationship there la between these stated objectives and program 
approacliea, tachtiiques &,n& IfflplementatiDnp and whmt procedures the 
achool a^steia enaploya to evaluate Title I success itself* 

NIE 'a Analysis of Ifldivlduallg^d Instr-uetlo^n 

In NIE- s study degign, Piatrict Survey II ^111 "study the ef- 
fecti^emeis cf Indivldmllf ed instructiDn iii -the tiachlng of 
reading andinath*" 

Congress obviously is quite favorably iapreased liy the reeults and . 
potential reaultfi jlii individualized plans ae the •corrterstone 

of individualized 'last ruction r In the 1974 educatlom aroendmenta to the 
E,S*E*A,3 In fact J taction 141 of Che la^ was amended to provide that 
'It is the Intent oi t >?here feasible^ the 

deyalopment f^r each educationally deprin/ed c^ild. ,.of an individualized 
wrttten educational plan Cmaintainad and periodically avaluated)^ 
agreed upcn ^blntly by the local educaticaal agency, a parent or 
gwrdlan of the child, and ^hm approprlaCea the chxld*" . 

KTe coffliBend NIE for the extensive my in ^hlch it* has been alert 
to this special Cougres^lonal imttrest and hms Mppad studies to 
^2£^miae thlj approach* 

Individualized Instruction obv-ioualy is not '*one to one,'* Rather ^ 
mys UlE, the Key characteristic of Individually tailored inatruction 
i^ ■'student progresn and remedial instruction ^hlch is based on student 
p^tfornianee on thi^ progress tests and allo-ws students to proceed at their 
owil rates," Strtndsrdigad instructipn has a class speed which reprasente 
a norm C t^ith tha brighter youngs ters either given special aisignments 
or beccraing borEdj and with the lagiing youngsters getting only the 
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attention^ the taaclier can spare) , Thm individualized instruction of 
Interest to UIE is to compare eacli student's skills' (usually In 
reading #iid matli) n^t against a alass norm, tut against a '-atandard 
instrument" which Idtntlfies deviations In learning rateS| calling 
for individual learaiag plans which art then carried out to the 
ability of the teacher In that mamm class group, 

NlE*a conttactDr under this award will ermine: 

Instructional tachtitqueg 

How compensatofyT3uca"tlon activitieg are Goordinatfed with 
regular 'lastruation (such cQor dination baing assumed desirable) * 
Is it sepatrate and "remedial*' ? Or is compensatory education lacre 
subltmlnaL and system wide ? 

What syateo! (If any) of stated performance objactlvas does the 
district employ ? 

The goals include deterniiilng the aspects of Individualization that 
ax^ most eruclalj assessing rnethods fox inv-olving parents^ determining 
conditions for succagsful Iraplemimtation. 

The contractor also is advised that NIE la not IntWested In the 
"unusual*- classTOQm -with pioneeririg work in it that too often depends 
upon the unusual teachers, (AlthDUgh another section of the study 5 that 
which seeks lnno\ratli?e practices and programs, certainly will depend quite 
heavily upon finding the unusual clasirooiH with pioneering work) * 

EKamlnatlgn of ^LlQcatlon of Funds 

Research on Allocation of Funds , the second major category of 
research^' seeks to "determine the results of possible changes in federal 
regulEtiotia governing local school' districts' management and delivery of 
compensatory educaticii services under ^Itla I, " 

Notwithstanding tlie research objective of testing both concentration . 
and diffusion of available funding as deviations from present 
regulatlon--aet practices, It^ is interesting to note that all 16 
districts selected for this experiment-phaie, NIE raportSs have stated 
in their propDsals that^ If accepted and given freedom to deviate 
froni normal HEW rulea^ they would disperse the funds far more 
(that la uBE the funds In a chlntied out vay to provide services 
to far morfiiof the total number of dlsadvantager. students in the 
district) . 
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This Is a rtault whose waanlMg NO should probe and not 
merely accept. Dees it mean that the diatc lets untfornLy 
belleye that the n^my to improre p^ograin af f activ^eneaa Is 
thrQugh le&s escpasidlture pM targat pujtlj spreadliig the 
program Impact ovar more pupils? Or m the othar hands Aomm 
It reflect laeaL polltioal and oparatlcaal preasurea to try to 
help ae maay ^^oungBteri as posalble? ^ Or fttlthsrs or bpth? 
Uniform ag it iss the dlatrlet regpons* pat term Gsrtaloly holds 
soma algnlflaaftci^ even as that pat taTa tendi to vitiate the 
original ia teat that the deia©natratlong test both ecncentratlon 
. and diffuaiitt ©fi funds frcm the preflent normml pattern* 

Also p while the work^stmtOTant .doHis not specifically' call 
for itp we bajlive it Ijipprtast f or iffeative aoet^analygis that 
the study CTOrdljfldtor ia each school be aeked Idefttlfy all ^ 
the major Title I I.S^E*A. dollars wrtilah proportionately end 
up aliocablf aad attributable to gli^ea cci^ensatcry progruis^ 
difficult am this irill be beaauBi 6i co-jilngllng . 

Other Costs oi Kote 



In additii^fii HIE has granted another series of awards for 
supportive studleSp fom awards are far studies &i "^Lteraatlye 
Dasigna t© Current Iractlae in Ce^ettsatory EducatlQn*-' a moat 
dsslrabls step. Others comissioa ettidies of progrM goalst 
operating cbaracterlstlca and pAer lutters* 

We urge that in every study there be sa^asla upon Identifying 
keya to aucaesa, Vhat worW that there n&t be 

eKmination of the sohool and @f aebocl programs jji Isolation^ 
but inoiuaion of eo™g»lty*s edueatloiial inject and value* 
For if soelp^eeono^c faetors not? ara a drag upon laamlng^ should 
they not be a focus of reaMrch sttsatlon as wah as is the elasaroMiT 



Sole Sonrae 

%All of the main awards for research usder this broad progrm 
were by co^ettti^e bidg esccept one*. 

■ ^ OA0 of the coiitracts KII Iseuedl Atk this work^progran waa 
on a sole source basis to the lawyers Connittee for Civil 
Rights Under Lw. rhls organlsatictt periodically engages in 
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litigation againit various agsnclei aueh as state educatloxi 
dapartma^Ci and echool districts* Their contract is for 
rasearch into state--by--s tata legal standards in the adminiacratlon 
of compaiisatory education, 

. One must question thli arrangement , By the awards Nil haa 
inhibited an advocacy group . One also must queitioti' the basic 
conflict in tha concractor'a action accepting such an award, 
with its stipulations against use of thr research fiftdlngs in 
actions against «a4ts" of government when such legal action ii the - 
Conanittee- s frequent practice, 1- 

Certainly the agency's justification for making th la awmrd 
sole^source^ uncomi^atitivBlyp should be public. Tha Council has 
baen provided vrith a copy; the Conncil renialns conceriiid about 
the propriety. 

Sifflraary Observation 

Our main cone lui Ion i abouc NIE's procedures in inltiattiig 
these studies arai 

0 That the q^uallty of the study program hae been reduced 
as the result of the short time= frame available to 
engender school district responses and cooparatlon 
that need to be Teprasentativa* 

0 That with the daBonstrationa there should have been 

a more deliberata if fort to select districts in a oanner 
producing repxesentat^l^enesSs rather than relying upon 
self ^selection or state selection with consequent bias. 

0 That all major contracts should have been cbmpetltlv^ly 
awarded, on elenietita of a fliatter as emotionally charged 
and complete blb compensatory education's avaluatlon* 

These shortcomlnga certainly are not all attributable to 
Congresi, after all, t^aa a major factor In the tight ttmetable and 
axtansive evaluation scope, Nevertheless , the problaois point to 
the need for improved risearch management across the boards from 
the legislative to tha action stagaa. 



^ One must 'keep In alnd a healthy skeptlclani as to i^hether- the 
answer lies la programs at all. 



VWh€ti the District I survey ins trumenta take final form, 
%je urge NIE to Mke certain that those instruments covmr 
^hat have bean suggested elsewhere as factors In the delivery 
of Improved learning rates among the disadvantaged* 

class size (smaller the better) 

school sl^e C^roall alao Is bettar) 

teacher eKperlance - 

teacher ability and preparation 

f afnily/rtelghborhood cultural Wrlablea 

year \roxind influence 

Gontinulty year to year through school 

cltiaens- involvement and responsibility 

CHow^ the educational syscem Involvea parents, and 

ho^ Che parents parcei^e themselves as having role 

and responsibility) 

What about aelacting schaols In differing types of nelghborhooda 
so as to aliow and control for the non^-achool envlronffleritg? 

To the extenc that Nil and OH atudlas do not define and 
examine true alternatives it becomea very important for the 
literature search to discover as much in tlie way of inaglnative 
experimentation as can ba found and described. 



Other CQiigaeiit.i 

The background literature and analysis saeni to pass over 
certaiti othar ma^or Issues i 

1* Hw to assess supplantatlon, if any, (Queried about 
tliiSi KIE said that this only could ba discerned through audit 
aTrangeEttentaj but If so why not seek that audit with 3--4 
cooperating districts to test whethar this appears to be a 
pirobleTO vorthy of raore attantlon later?) The Council recotmends 
that the ne« ^itle I regulations must define supplantatlon 
clearly, DHEW issued these regulations in draft formi for 
consent, Harch 11, 1975, It is highly regrettable that DHEl^ 
still has. not finalised chase Important guidelines, 

2, ¥hat oari be done to aompa:p0 pragT&sa by studentB 
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given aomp&nBatoTy education and groupB of similai* baakgraundB 
not given Buoh spmqial aid? Thi& i$ ajHtiaal to the und&Tlying 
asBwnptionB of Title I» . , 

3.; . .There should be a clear direction to the contractors 
examining Title 1 pragranis Cespeolally under Survey I) to 
develop data comparlag the level of education, eKperlence 
and skill of teachers working with disadvantaged children 
, vs.: children of more ■ affluent backgrounds in the same systems, 
frequency of substitutes^ and otTier such data to asaees what 
the school systeni "nDrmally*' malc#s available per disadvantaged 
youngster vs, other children, 

.4. Is the compansatory education special aid being un-done 
and negated by other activities and practices within the 
school system? ^ , _ 

5. How can there be aggreaive exploration of non-school 
alternatives? 

6, The study raust also require attention to the Isiue 

of scale — and by this wre do nob mean the size of the claiarooift* 
To what degree have echools at th# elementary level beoocje larger 
and better or worse? Is there a correlation between Institutional 
scale and quality in compensatory education? Is there a aorrelatlon 
between scale and pyramided overhead? 

7* In fact J what does the compensatory education dollar 
buy In various systems? ^ 

8, The Hawthoriia effecti it is disturbing that the evaluation 
approach starts from a zero data base and will rely upon a future 
two year study of aalect dlatricta. starting data collection 
after the districts have learned thaX"they are select. What 
allowances and corrective can there be for this? 

9 . Congress V aacpactatiori is the the NIE study results be 
available to guide consideration of new legislation as of 
September^ 1977^ wayp although necessary^ be too optimistic* 

Beyond these issues of projeat administration there lies 
a larger question. The larger significance of evaluation lies 
In asseesing the efficacy of doing anything at all. In this lights 
we believe that study of 'he effec tivene ss of compensatory education 



over a lonp Mm Is wore valuable as a measure of program 
tlcacy than tjliA ^fcatut^^ required NIE studies of program 
administra^ioin^ TOd of the factors behind appar ant variance in ^ 
school diatrliC^te ^uccegs wlCh compenaatory iduw^ And this 

lonoer term M^Wment of efficacy is the task of-the second major 
evaluation pregmmp that of the Of f ice of Educatlori (OE) . 
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PART III 



THE. OFFICE OF EiaCATlON ' 
RESEARCH PLAN FOR LONGETUDINAL STUDY 



/The Office of Education also is embarking upon an amblClous^ 
research agenda and plan, to aKaraine the effectiveness of com- 
penaatory education longitudinally- ovK yearg* OS's coxitYactDT 
W^lll "track" the educational progress of a larga number of dtsad- 
vantaged young people over five years, i^ith^a siKth year a possi- 
bility, 

While scope of the original study design now Is being halved, 
the teault of Congressional concern for Its cost, our analysis 
1^ based upon the original plan s lace as of this data the design 
of the reduced study has not been finally determined, OE ad\risea, 
X^t alone cotninunicated Co us* 



l&iatriiments and Approach 

As originally contemplated, the longitudinal apprQach wouLd 
start t^lth the developnient of a sample comprisad of sonie 5,O00 dla- 
advantaged young people no^ enrolled in early primary gradea, The 
aotitractor then would follow the progress of these indlvlduala, 
developing a comprehenaive data base about their educational e3C^ 
posura and success. 

The sample will not be limited to students receiving speciai 
C;ompensatory education that Is federally financed. In fact^ the 
nuftnber of individuals in the sample vrlll be of sufficiant maginL^ 
tude to perintt the crosa-'COmparlsons ME \^111 not be able to pro-- 
diicej andong and between those in y^arylng basic situations vis a vis 
eoiTnpensatory educatloni — 

Students whose education has been financed through 
Title I funds only; 

Students whose cost-^of-educ^clon involves not only 
Title T but state and local funds as well, altogether j 

J* 

Students who J although disadvantaged, are in schools 
and programs that only involve state and local funding; 



Students receiving no formal compensatory progratn. 



It. is helpful to syminarlge the mairi tagl^s of the CEstiidy: 



1, From a national scrapie of 5^000 s chooLs, get 4O0 to agree 
to participate. Load this sample of sclioola itfitli an €5€tra pro- 
port Ion of Inner-clt^ ichcols to compensate for the fact clig^c, his- 
toricallyi these schools vlll te low In tlieiic teBponse to a request 
that they participate. 

2, Develop a battety &f data coiLectlQn iiistriiinents oa laatters 
including 1 studetit background ,stiidexit coiiditlcni aiid healch s a£adent 
attendance and progratn participation, Ilodlfi? e^cisCiEig or new In- 
strumehte to free thern of biases bs niich po^elfala^ 

3* Fleld-tegt ptoceduris In some 20 schools, 

4. Train local coordlaatoxa, Includiai tTainiiig thenn in 
test admlniatratiorL and suri;ey inaBageiriant * 

5. Make cost effiactivaTiesj studies of various approachas. 
6* Make a patc Iclpatloii siibatiidyi 

7. Identify prograia successes and failures* 

8. Develop detatled sp€clf Ications of aCudy iasitas, Dev^eLop 
detailad data coHectlca and maaageTnent plaTi^ 

9. Monitor and inatiage the &n-gQltij acti\^lties, 
10. Prepare reports and otlier s^tudj? prodiaccs , 

The Council ReaGtlons to' the OE Regeareb TLan 

Our coinmints regarding this study plan are limit €d ty crc 
factors. The stud^f-g final ton and du-rstlon aie mcertVla* 
And we have not yet recel^^ed for ra^le^ a ccpy of the coptract 
involved^ which presumably ^ill different and loore detailed 
than the Request for PtopoaaL <RTP) , 

With these caveats, , our r&^±etg has fiotinid tJie OE approacli 
essentially sound. ^ -HfO'^cs^ do have several Teaer^atloiis ^hlch 

even at this date Itivolvc polTiCs of approscli and &t mcChiDdoLogy 
which can be improved* 



1. The horn* survGy is all-important and must be given 
special attention by the contractor. The home survey, in 

fact, should reveal far more about readiness to learn and ability 
to learn, than will be derived from observing and assessing the 
individual students in the school. 

As now articulated, the home survey appears to ask the 
contractor to emphasize assessing economic status of the family. 
This is far too narrow an aperture through which to gee and 
assess factors relevant to learning. The OE Work Statement does 
recognize the soc io-cultural factors; what we ask iethat there 
be emphasis upon analyzing them. There is some correlation be- 
tween poverty and a poverty of educational interest, but by no 
means a direct and dependable one. 

2 . By now, government should be well- tuned to the Impor-^ 
tance of drawing survey instruments broadly assessing the socio^ 
cultural environmeint and context, in evaluations. asseSBlng the 
impact and efficacy of institutional intervention. As one ex^ 
ample, we point to the on-going longitudinal study OEO started 
5-6 years ago, "tracking*' the status of some 5,000 poor and near- 
poor families (a study continuing under DREW auspices)* One 
criticism of this study's design has been its underanalysis of the 
socio-cultural factors in economic success and failure* 

We urge strong focus, in this new OE study of 5,000 dis^ 
advantaged youngsters, upon development of comprehensive Infor-* 
mation about the soc io-cultural background of the S^OOO families 
from which these 5,000 young people come. For if t}m ctudy do&(J 
not ^^Gontrol" for the non-school environment there^ is no way th(? 
study aan -assess school progra/n effectiveness (a point with rjqual 
application to the M IE research plan revieioed earlier'). 

3. We believe that the OE work statement tends to under= 
recognize the problems that arise due to the frequency with which 
family moves uproot families from neighborhood and school. The 
annual rate of moving in our society averages one faniily in seven. 
Cumulatively, this then is a major problem in any 5-6 year 
longevity study. 

It is one thing to maintain sample size by substituting 
other young people for those who move (which OE asks its contractor 
to do) ; it is something else to maintain sample representative- 
ness. After all, it is logical project that over 5-6 years 
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a large proportion of the original sample will h^ve moved t Be- 
-sides, is the sample really representative If the contractor 
always is removing from it the students who movei whose ex- 
perience of educational discontinuity is an important part of 
the real world (and probably an important factor in lagging 
learning rates)? 

4, The aontmatoi* is asked to feeep a log of mah student's relevant 
eduQational exp&vienoe and pajntiaipation. Fine, This log should 
enaompass all sahooi aativities^ however^ not merely the student's 
Gxposnre to aomp&nsatory approaahes and 8y8tm3. For' of what 
value is it to reaord and analyze the effeat of a -supplgrmnt'' 
without reQording and analyzing as well the student's basia 
eduaationai ''diet'' to whiah it ^a supplement? Of partiau- 
lar .importanae^ we Buggeat^ would be a base-^mapping of the 
student's linkages and peer group. To what extent am auah net^ 
.works be mapped as a step toward assessing whether the sahoal 
peoaess is in faat reaahing and turning-on the youngster by ' 
rmahing the value base on whiah his premiaes for aation (e.g, j 
hia interest in eduaation) rest? - 



We also believe that the OE study muat be of a duration longer 
than the 5-6 years now envisioned,' Our reasons are aignifiamt 
enough to be set forth under separate heading^ dealing with the 
issue of Effiaaoy , 
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PART IV 

GENERAL GO^IENTS REGARDING EFFICACY 



Any major evaluation of a public program and policy must ad" 
dress issues at two levels i 

Questions of technique , and 

Questions of efficacy. 

Both the NIE and the OE evaluations seek to assess effective- 
ness* Neither study will do this . Both focus upon performance 
under present conditions only* Effectiveness has yet to be well 
defined* 

No norm or standard is provided| in fact, a major objective 
in both research programs is examining present performance under 
present conditions with NIE focusing upon institutional per= 
formance and with OE focusing upon achievement gain by the indi- 
vidual* 

When effectiveness is measured^ as _well ^s the standard by 
which it Is measured 3, constitutes a critical Issuer It is our 
opinion that a major difficulty if not weakness in these NIE and 
OE efforts to evaluate Title I^ E.S.E.A*^ is their eKpectatlon that 
effectiveness can be measured in terms of Improved rate of advance 
at the time ^ that Isj improvement in cognitive skills durin g 
the spec_ial^ augmented aducation Title I makes possible i. 

The Congressional mandate for the NIE studies specifically 
asks for assessment of effectiveness. We prefer to interpret 
effectiveness as carrying strong implications of a change that 
endures: that iSs a change for the better in the level of cognitive 
skills knowledge and interest which Title I students carry from 
school into adult life. 

The experience of the adult remedial education programs em^ 
bodied in such activities as those of the National Alliance of 
Businessmen (NAB) provides a relevant example. When NAB started 
sponsoring in-plant worker remedial education In 1968, many of 
the illiterates and near-illiterates they enrolled were people 
whose childhood educations once carried them well into high school 
and even college • In factp' the pioneering Texas program on whose 
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experience much of the NAB program was baaed involved some 80 
wprkers testing at. below fifth grade level. Of this groupv a — 
third were high school educated; five even had been to colleaa 
(but lacked degrees) . 

A major educational problem which schools alone probably 
can-do little to overcome is the affect of a life-style in which 
^ one s work does not require extensive verbal and mathamacical 
skills, coupled with the electronic age in which there usually Is 
small reliance upon newspapers and books. In other words, what 
the schools have to contend with is a life=styie in which the 
paraphernalia and reinforcements that maintain educational 
competency through daily experience are. for many, weak or non- 
existant. 

An examination of effectiveness only can be undertaken 
over a span of time, that is, longitudinally, and with well-constructed 
control for all other variables, especially cultural, social 
and aconomic. Examining shorter term manif estat 'ons of effectiveness 
runs the great risk of being misled by transitory changes soon 
overwhelmed by the other and more Influencial factors which the 
school program does not substantially Impact upon, let alone 
control, especially home and environment. 

Of course, If such an erosion of advantage would be discovered 
(and there is enough evidence of the probability in the Head 
Start experience alone to justify looking into It), this would 
not imply fault in the process of education as much as the danger 
in expecting education alone to overcome the negative factors and 
influences schools themselves cannot control- or offset. 

The iraplications of these comments, for both the NIE and OE 
study designs, really run in two directions. We see nothing- in the 
NIE design which really assess efficacy in the long range manner 
we believe essential. It Is much more a study in the effectiveness 
of process than of program effectiveness. The OE study 
design will collect data for three years on a sample of youngsters 
now in the first six grades (thus, for those in the sixth grade 
the first year of the study, following them through the eighth) . 
OE suggests as. an option to be determined later, that of following 
classes through the ninth grade as well, in a fourth and fifth 
year of the study. This and more should be done. 

30 
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Significance rei QE Study 



We suggest that it should be possible to aKCend tha study 
frame ^ without added ag^nge^ by rearranging the sample* For 
example^ it probably is of little value to measure the attainment 
in the study's third years of youngstars In the first grade 
who will not be assessed again thereafter; why not shift this 
commitment of funds Into; more fodus upon measurement of 
achievment levels at higher years (comparing Title I allglbles 
who have been receplenta of Title I special attention and Title I 
eligibles who have not)? 

Even while exploring all such possibilities for lengthening 
the years studied by OE5 we also suggest two other steps that 
might well telescope the activltyj bringing savings that 
make extension possible 1 

1* Studying compensatory education levels every other 
year* deriving the Inbetween achievement level through 
extrapolation* 

2*. "Mining" existing records Ce*g*, as NIE is endeavoring 
to do with the Rhode Island records that reportedly show 
achievement levels for the past ten years # 



PART V 



COORDINATION 



Question has been raised as to the overlap , if any, between 
NIE and OE plans to analyze and evaluate the effectiveness of 
Title I compensatory education approaches* 

In this regard, both studies aim at analyzing the fundamental 
pufpoees and effectiveness of compensatory education, approaches 
and programs J and therefore they should have significant cross-= 
overs* Regrettably^ in our judgements they have been avoided 
by the agencies i 

Coroments Regarding Coordination 

The focus of coordination between the agency staffs 
concerned for these two scudles seems to be at the procedural 
level J and even that must be adjudged weak. While the two 
staffs make sure they literally do not get in each other's way, 
coordination of a positive sort seems of a low order, 
EKamples For months, OE did not receive copies of the NIE RFPs- 
and apparently did not feel the need to even ask for them until 
a periodic review session September 4th surfaced the point. 

In addition J we question the decision by the NIE-OE project 
managers to deliberately avoid including any of the same cities 
in their studies J in the intemst of minimiBing the bidden on 
aoopBmtiny eahaol ayetems^ yet in con^ination these two studies^ 
while especially involved^ represent but a small part of the 
annual burden of studies and reporting in school districts 
most of which is mandated in order to receive funds* It would 
seem that there should have been a deliberate effort to build 
bridges and crossovers between the data sets these two studies 
will present. 

The need for impramd aoc^dination aatually go&a all the way 
baak in the anatony of th&ee ttoo atudi&B^ to the faat that NIE 
md OE report to different 8i^-aomnittee3 in the BouQe of HepreBentatives 

The Council also is concerned about another weakness in coor^- 
dlnation and contact which our study has revealedi there appears 
to be a tendency among those commissioned to evaluate a program, 
to do so with minimal contact with, the program's administrators* 
It appears that NIE has not made sufficient effort to extract in- 
sighta and information^ if not also data, 'from the racordg and 



depth of staff Insight to be found in the DHEW office admirils'- 
taring Title Ij EiS.E.A, That office has ten years' background 
in this program. While indepsndent evaluation has the advantages 
of fresh Htaights and objectivitys it still makes sense t-o 
glean as much as possible from the aiders established cencer 
of eKpertiea about the Title I program* 

2%e CoimaiZ reaorm&ndB that eBpeaialty at this tate dat#j 
the aomp&nBatory eduaatian progrm! aAninietratoPB in DHMW be 
dmply involved in advising the OE and NIE TitlB I evaluationB 
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CONCLUSION 



EVALUATION is an aseential elament of the affective and sus- 
tained upgradiiig of program performanca* 

The practice of itj however^ requires great cars. As wa have 
tried to point out here, out of the OE and NZE efforts to mount 
major evaluationE of Title E.S.E^A, the government can learn 
some valuable operational lessons for this and other occasions: 
the need to plan well allowing ample time to carry out the plan 
well; the need to coordinate,^ the need to derive maKimum 
value from existing sources and records, the need to insist on 
repreaeMativeness, the need to structure the evaluation so that 
it really measures lasting impact, the need to imaginatively con- 
sider alternatives, and the need to truly welcome and involve 
public participation* 



This is what good government is all about. 



PEMINm EXCERPTS PROM P,L, 93-380 
EDUCATION AMNDlfflNTS OF 1974 



"Bk(% lAK (li) Tho Omimiimionf'r ftlmll provWo for iiulnpnnilniit 
nvalnutions which «lp,^cnbR and inpMiiro the linpftct of pro|ft'njnB ftiul 
pro}mtfi fu^isNf iiiiclpr tliifl tMh. B\%v\\ cvfthirtMoim mfty pmylclofl by 
noiitrarfc or oMmr arrnngcmmita, unci nil airh ovfiluiiUon.^ slinll iimue 
by compnNnfc and IncleponclGnt porsoTiii nnd rtinll Innlinlr., wlinnnvrr 
po^iblni opIiilonR ^tnjUrcl fmm progrnn! or pnijonl* piii'ticlpniita 
about f hc\ ^1 rtyi^Uii mid of md\ prc?grninH nr pnijpnla, 

-*(b) Til? < oiiiiiiWoiuM* Nhall dcvrlnp iiiit! jMibHHh HhiiiiliirdH for 
ovnliinfioii cif progriuii nr prnjort ritt'c'liviMims in nrhirviiig tho 
obj;VlivrM <»r tbiN jitip, 

'rii^ C Niiiihili^iiinrrNhiilU wbfMv iippriiprintis (*onHiilt with Hinir 
iiginii'irH liViii'ilpr hi provnU* for joinlly HjM>ijHnri*!l libji'i'l ivo ovnbinf ion 
RhiflioH <>r prn^nitfiH iuh! prujciMH ii^tHirtl iiiuUm' Ihin ti(h> wtlhiii \\ 
HMO. 

"(d) Thi* (^iiiiniiKHloiuM' MhiiU pvovido iaHliiit^ odurnf ioinil iipi»n(»H*Sj 
iiuhIi'Ih fill' ovjiiiinlHiim of nil prif^iMuuH mMidiirliul uiidpriliiH iijlOj for 
llinir !iH^ in nirijMUir tinl Ibi'ir fiinrUnim liinh^r Hi'Hion t UlliO^ mnvh 
nlmll iin'liitlp uniforin iunmlnn'** itiul j'rili*i iii lo lu' iiHIUimI by loi'ld 
gduentional dgenclot^ ns tvoU &i by ths UMb agm^ in tho evaluation 
of such progrftrtiB. ^ 

**(e) riitt ConiniW©ner Bhall otovidft iUeh t^hnical and othnr 
ai?8lBt^nc© as nmy ta iircc^Bry to State edueatfonal agencies to onable 
thoin to asiiit j@€a1 oducational agenalM In tlm dereldpinont and appll^ 
cntioii of a syat^iufltic avaltiafcioii of progi^ams ki ae^srdanG© with tha 
inmleli clcvDlopcd by thft (^mmi^ionon ; ; ; 

"(f) TJio inndel.^ dcvelnpad by tho Oomiinf^loiior Hball Hpucify 
objnctlv6 criteHfl which ihaU hi MlimQ. ift the bvaluation of aH pro^ 
^i-nnm m\A Hball oiitHne teehniqua^ (itiob aa lonaitiidlnal Btudic^ of 
I'bildrnii Inynlvnd lit mvh pniprainH) and niRthodomjiy (mifXi m\hts\m\ 
of imiH which ylnld enjuparable rssultM) for prodiioiiig data whieh aro 
(rnniparnhlo on n- BtatrAVidB and nationwldo baBm. 

"(k) T^bn CoiiiiniK^innnr hIuiII inahn a rrpoi tlo Mm rnspacLivn c*.oiii^ 
nilttcos of tliD Con^Pcw having Icgiilafclv© 3uriBdiotioii over pi ogrnjni 
aiithori^nd by fclils tstlD and the reapecfciya Conimittcoi on Appronria' 
tSoiii cojiceniing hiB progre^ in chiwing otit this section not latGr 
tiian January 3Ij 10?5j and thereafter lib slmilrnport to suGh comniit'' 
iecH no hiWr than Janiiary 3t of ennli calniKlar ypar the rc^^ulla of tlie 
yvalnntioim of prograiiiH and pnjjnc^U^ rCMpiircfl niidor thin Hrction, 
wb id » il I al l l>e com p vei ienBi ve and clGf jii led , as iip-t Q==d atd na poBHib la, 
and bused to fclio ina^ciinnm oxtent pwiblo on objective nioasuraiuont^i 
ifi^clber with any other related findiiigH and ovahmtiouMj and lii.^ 
riHuiuunciidfttJouB with rwpecfc to leg^ 

"(h) The CorTiinl^ioner shall nlRo deVidop a syat<*nj for the pother' 
ing aiul dtH.^niiiiatiori of roHultH nf ovabiatiuUB and for tlio idnntiflcn^ 
lion of e.Nciiiplary pi*ojJrranis and pro]«!C<^45 or of pai1.inularly eirnutivo 
plenmntH of pvofiianis and prpjcct^, and for tlie diMSnnunation of 
information conCGrning such progranis and prejecta or such elenienti 
ihei eof to State and loeal eaucatlonal o^Dneia^ rc*Hponaiblc for the 
design and aonduet of programs and projects under this titla, and 
to flin mincatinn profession and thej^neral pubric* 



""Sec. 417, (a) (1 ) ^*ot later than Noveniber 1 of #nch vear. the Serra^ 
■■ - tary sliull h^nmnir to the romroitte© on Kclucntion iiii^ nnbor^^ 
Jfoiiie of Repra^ntiitives and the CwiiniittGe en Labor and PiibJir 
Y fifarp of the Senate an anmial evaluation report which evaluntes 
t\w offectivene^ of appjicabk programi in achieving their leHshitod 
inirposes together with recomniinaations relating tp such pro^mms 
for the improvement of euch programi which wilt i-eiiilt in greater 
etfer rivenesa m aehi&ring such pui^^«. In the caj# of anr evahiatien 
repfiit eviilucitiii| spacjflc proems and projects, such Wport shall— 
(A) m forth goaU and specifio ©bj^tivw in quaihatlve and 
miantitative termi for all programi and projecta agisted imdfr 
the applicable program ^ncemed and relate thc^ ™ls and 
ob^oetives to the purposis of tueh program ; 

'*{B) contftin infonnfttion on th© mogrm being mad© durinff 
tjie pi^viouB fiscal year toward the acliieviment of aueh leoals and 
objeetires; - - 

;*(C) describe th# cwfc and benefits of the applicable program 
hein^ evalusted during the pmvioui fiical %^ear and identifr which 
sector of the public mcmv^ the benefits of such proffmm and bear 
tlw eosti of siieh program ; . 

■HD) contain plahi for implementing cormrtive action and 
rucornmtndations for new or amended leriilatlon where war- 
rantee!; : 

''(E) contain a liiting identifying the principal analries and 
itiuliei supporting the mftjor concluiions and Fecwnmendatlons in 
the report; and 

**{F) be prepared in concise summary' fom with nece^ary 
aetailed data and appendli^ 
•*{2) In the cast of programs and projects a^iited under title I of 
the £lementai7 and Secondaiy Education Act of 1065, the report 
under tins subsectfon shall include a survey of how manT of the chil- 
dren counted under section 103(c) of such Act pa^eipate in such 
programs and projects, and how many of such chUdren do sot, and 
a survey of how m»nj educationallj diwdvsntaged chlldrm partici- 
pate m such prMfams and projects, and how many aducatlcmally dii- 
adranta^ cliiiaren do nrt. For puqp^ of the Blading sentence, 
the tem wtucationally disadrantagedrchildren^ refers to ehndr^n who 
arB achieving one or more ye^ri behind the achlm'timent COTect^ at 
the appi^prtateg^de level for such diildren. 

"(6) Each ev^tustion report submitted pursuant to subsection (a) 
ihall contain I (1) m brief d^ription of each contract or grant for 
e vajimtion of any program { whether or net such contract or grant was 
made imder stetson 416) any part of the perfonnance of which 
P^i9^ 1 "® preeedinff year, (i) the name of th© firm or 

mclividusl who IS to out the evsluation, and (3) th© amount to be 
paid under the contrnet or g^LAt^ 



Sec. 82L (a) In addition to the other ftuthorities^ rtsponiibilities 
and duties conferred upon the,KaUonal Institut© of Education (here- 
inafter referred to as the ^•Inatitute") by seistion 405 of the General 
Education Proviilohs Act and nofcwithrtanding the second tentenco 
of S!iteection (b) (1) of such section 405, the Institute shall undertake 
a thorough evaluation and study of ^mpensatpiy education programs, 
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jnclndiiig such progrftma con Btatas and such j^rogrrasjBon^ 

ducted uiidor tilin 1 oi tlift KlRnicntA^ and Secondary j^dtication Act 
of 1 905, Such itudy ilmll iiachidfi— 

(1) an csaniinnfiou of t)m fundnmentftl piirpwa of such pro- 
gnnnii nnd ttici <ifri^cHvmioiB of siioh pro^ami in attaining' Bueh 
purpose; 

(2) an nnnljiis of moani to identify neoumtoly the clilldicn 
who hnvc tlic grimtcst need for rnich progrnnis^ in keeping with 
the fundanionlnl piiriH^^ thereof I 

(3) an aniiIyHm of Hie ^.ffeetivcnw of mgthmls and piwedures 
for Ttireting thn (uhirAtinnal neodi of ohlldmn^ hielndiiig the wm 
of indivldniilii^^d writtDn cdiicational pinna for childront nnd 
progrnmi for trnining tim tendicriof chlldnm | 

(4) an e^cplomtinn of nlteniattvo methodH, inchiding tlie i^sn 
of proc^^uroi to assRHs educations! disadvantag©; for diitributing 
fnndB nnder such programs to Stftt^ t:o i^tatci ndn&ational n^,n» 
cir^, And tci Inmil cdncatinnal agciicioM In an eniiituble and cfllicient 
manneri which will ncCiUnitely raflcct onrrcnt cojidltlonB nnd inmirc 
that Hnch fnndB rennh the nrmn of grcntcst cnrraut need and nm 
t»frr*ctivnly uHed for MUnh areas; 

(n) not more than W experimental progranis, which shall be 
rrfi^nably geographically itiprt^ntatlvo, to 1^ ndministni'od by 
tho Jn^^titiiiei in cn^^ whei% the Institute detormlnca thnt iuch 
e^cpcrinicntftl progniniy nr© neeo^ry to cnrry out the purposes of 
clanHT-H (1) fhroiigli (4)^ nnd the X^oinmlFHioner of Kdncation is 

uuthoriEed, notwitJiBtanding any proviBion of title I of the Jfllc- 
mentaiy and Secondary Eaueation Act of ip6fi| at the reqiicKt of 
the Instltuta, to appro ve the xm of grants which cdircational agen- 
cica are Dllriblo tu i^ceivo under such title 1 (in ciim^ whera the 
agency elipple for lUch grint agr^ to such use) in order to carry 
out such escperiniontal programs i and 

(6) fii^in0S and reeommendationsi including feeamniendationi 
for changes m such title 1 or for new le^lilations with i'OS|>ect to 
the niftttersstWied under elau^^ (1) through 

(b) TJie National Advisory Council on the Education of Disad- 
vantaged Children shall advise the Institute with raspect to the design 
and eKecution of such study/ The Comini^ioner of Kducation Mhnll 
obtain and tranimit to the Initlfcut© Hich information as it ehnll 
it^qucit with rapect to programs carried on under title I of the Act, 

(c) The Tnatitute shall make an interim report to the* rreHident 
and to the Congress not later than Dwrnljcr 3J, lf)T6, and shall nnilce 
a final report thereto no later than nine months after the dnte of suh- 
mission of such interim report, on the result of its Miidy nondncied 
under this section* Any other provlmon of Jaw, rule, or reguhitinn to 
the contrary Jiotwithstandingj such ro^^ not be snliniiilrd to 
any review outside of the Institute before their transmittnl lo the Von- 
greHJ^j but the President and the Commissioner of ICdncation niay ninke 
bo the Congre^ such recommendations with reipect to the conteiitH of 
the repoits as en eh may deem a ppropriate* 



